A FOCUS ON CONTENT: THE USE OF RUBRICS IN PEER REVIEW TO GUIDE STUDENTS AND INSTRUCTORS

Students who are solving open-ended problems would benefit from formative assessment, i.e., from receiving helpful feedback and from having an instructor who is informed about their level of performance. Open-ended problems challenge existing assessment techniques. For example, such problems may have reasonable alternative solutions, or conflicting objectives. Analyses of open-ended problems are often presented as free-form text since they require arguments and justifications for one solution over others, and students may differ in how they frame the problems according to their knowledge, beliefs and attitudes.This dissertation investigates how peer review may be used for formative assessment. Computer-Supported Peer Review in Education, a technology whose use is growing, has been shown to provide accurate summative assessment of student work, and peer feedback can indeed be helpful to students. A peer review process depends on the rubric that students use to assess and give feedback to each other. However, it is unclear how a rubric should be structured to produce feedback that is helpful to the student and at the same time to yield information that could be summarized for the instructor.The dissertation reports a study in which students wrote individual analyses of an open-ended legal problem, and then exchanged feedback using Comrade, a web application for peer review. The study compared two conditions: some students used a rubric that was relevant to legal argument in general (the domain-relevant rubric), while others used a rubric that addressed the conceptual issues embedded in the open-ended problem (the problem-specific rubric).While both rubric types yield peer ratings of student work that approximate the instructor's scores, feedback elicited by the domain-relevant rubric was redundant across its dimensions. On the contrary, peer ratings elicited by the problem-specific rubric distinguished among its dimensions. Hierarchical Bayesian models showed that ratings from both rubrics can be fit by pooling information across students, but only problem-specific ratings are fit better given information about distinct rubric dimensions

Redesigning educational peer review interactions using computer tools: An introduction

Peer review is a family of instructional techniques. Historically, these have been employed in writing and many other educational domains. Modern computer technologies facilitate the use of peer review, which is especially relevant to educational settings where it is not practical to administer peer review manually. The use of computer support for peer review has shed light on many important scientific questions, some of which we summarize. These findings set the context for the papers in this special issue, which demonstrate how computer support for peer review enables research on peer review itself and on its pedagogical significance

Teaching Case Analysis through Framing: Prospects for an ITS in an ill-defined domain

Abstract. Intelligent Tutoring Systems research has made assumptions that may be violated in illdefined tasks. We describe ethics case analysis, an important educational yet ill-defined task in the domain of bioengineering ethics. We discuss an ITS approach for this task that involves structuring the learning experience and using AI to help guide student peer reviewers